{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Credit Scoring in FHE\n",
    "\n",
    "In this notebook, we build and evaluate a model that predicts the chance that a given loan applicant defaults on loan repayment while keeping the user's data private using Fully Homomorphic Encryption (FHE). It is strongly inspired from an [existing notebook](https://www.kaggle.com/code/ajay1735/my-credit-scoring-model) found on Kaggle, which uses the [Home Equity (HMEQ) dataset](https://www.kaggle.com/code/ajay1735/my-credit-scoring-model/input). In addition, we compare the performance between the original scikit-learn models and their Concrete ML equivalent. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "import pandas as pd\n",
    "from sklearn.ensemble import RandomForestClassifier as SklearnRandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression as SklearnLogisticRegression\n",
    "from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Import models from scikit-learn and XGBoost\n",
    "from sklearn.tree import DecisionTreeClassifier as SklearnDecisionTreeClassifier\n",
    "from xgboost import XGBClassifier as SklearnXGBoostClassifier\n",
    "\n",
    "# Import models from Concrete ML\n",
    "from concrete.ml.sklearn import DecisionTreeClassifier as ConcreteDecisionTreeClassifier\n",
    "from concrete.ml.sklearn import LogisticRegression as ConcreteLogisticRegression\n",
    "from concrete.ml.sklearn import RandomForestClassifier as ConcreteRandomForestClassifier\n",
    "from concrete.ml.sklearn import XGBClassifier as ConcreteXGBoostClassifier\n",
    "\n",
    "CONCRETE_ML_MODELS = [\n",
    "    ConcreteDecisionTreeClassifier,\n",
    "    ConcreteLogisticRegression,\n",
    "    ConcreteRandomForestClassifier,\n",
    "    ConcreteXGBoostClassifier,\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the HMEQ dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reading the dataset\n",
    "df = pd.read_csv(\"hmeq.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean the dataset\n",
    "Further details about dataset cleaning can be found the [original notebook](https://www.kaggle.com/code/ajay1735/my-credit-scoring-model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace missing values\n",
    "df[\"REASON\"].fillna(value=\"DebtCon\", inplace=True)\n",
    "df[\"JOB\"].fillna(value=\"Other\", inplace=True)\n",
    "df[\"DEROG\"].fillna(value=0, inplace=True)\n",
    "df[\"DELINQ\"].fillna(value=0, inplace=True)\n",
    "\n",
    "df.fillna(value=df.mean(numeric_only=True), inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BAD</th>\n",
       "      <th>LOAN</th>\n",
       "      <th>MORTDUE</th>\n",
       "      <th>VALUE</th>\n",
       "      <th>REASON</th>\n",
       "      <th>JOB</th>\n",
       "      <th>YOJ</th>\n",
       "      <th>DEROG</th>\n",
       "      <th>DELINQ</th>\n",
       "      <th>CLAGE</th>\n",
       "      <th>NINQ</th>\n",
       "      <th>CLNO</th>\n",
       "      <th>DEBTINC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1100</td>\n",
       "      <td>25860.0000</td>\n",
       "      <td>39025.000000</td>\n",
       "      <td>HomeImp</td>\n",
       "      <td>Other</td>\n",
       "      <td>10.500000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>94.366667</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>33.779915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1300</td>\n",
       "      <td>70053.0000</td>\n",
       "      <td>68400.000000</td>\n",
       "      <td>HomeImp</td>\n",
       "      <td>Other</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>121.833333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>33.779915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1500</td>\n",
       "      <td>13500.0000</td>\n",
       "      <td>16700.000000</td>\n",
       "      <td>HomeImp</td>\n",
       "      <td>Other</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>149.466667</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>33.779915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1500</td>\n",
       "      <td>73760.8172</td>\n",
       "      <td>101776.048741</td>\n",
       "      <td>DebtCon</td>\n",
       "      <td>Other</td>\n",
       "      <td>8.922268</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>179.766275</td>\n",
       "      <td>1.186055</td>\n",
       "      <td>21.296096</td>\n",
       "      <td>33.779915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1700</td>\n",
       "      <td>97800.0000</td>\n",
       "      <td>112000.000000</td>\n",
       "      <td>HomeImp</td>\n",
       "      <td>Office</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>93.333333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>33.779915</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   BAD  LOAN     MORTDUE          VALUE   REASON     JOB        YOJ  DEROG  \\\n",
       "0    1  1100  25860.0000   39025.000000  HomeImp   Other  10.500000    0.0   \n",
       "1    1  1300  70053.0000   68400.000000  HomeImp   Other   7.000000    0.0   \n",
       "2    1  1500  13500.0000   16700.000000  HomeImp   Other   4.000000    0.0   \n",
       "3    1  1500  73760.8172  101776.048741  DebtCon   Other   8.922268    0.0   \n",
       "4    0  1700  97800.0000  112000.000000  HomeImp  Office   3.000000    0.0   \n",
       "\n",
       "   DELINQ       CLAGE      NINQ       CLNO    DEBTINC  \n",
       "0     0.0   94.366667  1.000000   9.000000  33.779915  \n",
       "1     2.0  121.833333  0.000000  14.000000  33.779915  \n",
       "2     0.0  149.466667  1.000000  10.000000  33.779915  \n",
       "3     0.0  179.766275  1.186055  21.296096  33.779915  \n",
       "4     0.0   93.333333  0.000000  14.000000  33.779915  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove features BAD, JOB and REASON from the input feature set\n",
    "x_basic = df.drop(columns=[\"BAD\", \"JOB\", \"REASON\"])\n",
    "y = df[\"BAD\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Credit scoring with Concrete ML\n",
    "In the following step, we first define the scikit-learn models found in the original notebook and build their FHE equivalent model using Concrete ML. Then, we evaluate and compare them side by side using several metrics (accuracy, F1 score, recall, precision). For Concrete ML models, their inference's execution time is also provided when done in FHE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(\n",
    "    model, x, y, test_size=0.33, show_circuit=False, predict_in_fhe=True, fhe_samples=None\n",
    "):\n",
    "    \"\"\"Evaluate the given model using several metrics.\n",
    "\n",
    "    The model is evaluated using the following metrics: accuracy, F1 score, precision, recall.\n",
    "    For Concrete ML models, the inference's execution time is provided when done in FHE.\n",
    "\n",
    "    Args:\n",
    "        model: The initialized model to consider.\n",
    "        x: The input data to consider.\n",
    "        y: The target data to consider.\n",
    "        test_size: The proportion to use for the test data. Default to 0.33.\n",
    "        show_circuit: If the FHE circuit should be printed for Concrete ML models. Default to False.\n",
    "        predict_in_fhe: If the inference should be executed in FHE for Concrete ML models. Else, it\n",
    "            will only be simulated.\n",
    "        fhe_sample: The number of samples to consider for evaluating the inference of Concrete ML\n",
    "            models if predict_in_fhe is set to True. If None, the complete test set is used. Default\n",
    "            to None.\n",
    "    \"\"\"\n",
    "    evaluation_result = {}\n",
    "\n",
    "    is_concrete_ml = model.__class__ in CONCRETE_ML_MODELS\n",
    "\n",
    "    name = model.__class__.__name__ + (\" (Concrete ML)\" if is_concrete_ml else \" (sklearn)\")\n",
    "\n",
    "    evaluation_result[\"name\"] = name\n",
    "\n",
    "    print(f\"Evaluating model {name}\")\n",
    "\n",
    "    # Split the data into test and train sets. Stratify is used to make sure that the test set\n",
    "    # contains some representative class distribution for targets\n",
    "    x_train, x_test, y_train, y_test = train_test_split(\n",
    "        x, y, stratify=y, test_size=test_size, random_state=1\n",
    "    )\n",
    "    test_length = len(x_test)\n",
    "\n",
    "    evaluation_result[\"Test samples\"] = test_length\n",
    "\n",
    "    evaluation_result[\"n_bits\"] = model.n_bits if is_concrete_ml else None\n",
    "\n",
    "    # Normalization pipeline\n",
    "    model = Pipeline(\n",
    "        [\n",
    "            (\"preprocessor\", StandardScaler()),\n",
    "            (\"model\", model),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    # Train the model\n",
    "    model.fit(x_train, y_train)\n",
    "\n",
    "    # Run the prediction and store its execution time\n",
    "    y_pred = model.predict(x_test)\n",
    "\n",
    "    # Evaluate the model\n",
    "    # For Concrete ML models, this will execute the (quantized) inference in the clear\n",
    "    evaluation_result[\"Accuracy (clear)\"] = accuracy_score(y_test, y_pred)\n",
    "    evaluation_result[\"F1 (clear)\"] = f1_score(y_test, y_pred, average=\"macro\")\n",
    "    evaluation_result[\"Precision (clear)\"] = precision_score(y_test, y_pred, average=\"macro\")\n",
    "    evaluation_result[\"Recall (clear)\"] = recall_score(y_test, y_pred, average=\"macro\")\n",
    "\n",
    "    # If the model is from Concrete ML\n",
    "    if is_concrete_ml:\n",
    "\n",
    "        print(\"Compile the model\")\n",
    "\n",
    "        # Compile the model using the training data\n",
    "        circuit = model[\"model\"].compile(x_train)  # pylint: disable=no-member\n",
    "\n",
    "        # Print the FHE circuit if needed\n",
    "        if show_circuit:\n",
    "            print(circuit)\n",
    "\n",
    "        # Retrieve the circuit's max bit-width\n",
    "        evaluation_result[\"max bit-width\"] = circuit.graph.maximum_integer_bit_width()\n",
    "\n",
    "        print(\"Predict (simulated)\")\n",
    "\n",
    "        # Run the prediction in the clear using FHE simulation, store its execution time and\n",
    "        # evaluate the accuracy score\n",
    "        y_pred_simulate = model.predict(x_test, fhe=\"simulate\")\n",
    "\n",
    "        evaluation_result[\"Accuracy (simulated)\"] = accuracy_score(y_test, y_pred_simulate)\n",
    "\n",
    "        # Run the prediction in FHE, store its execution time and evaluate the accuracy score\n",
    "        if predict_in_fhe:\n",
    "            if fhe_samples is not None:\n",
    "                x_test = x_test[0:fhe_samples]\n",
    "                y_test = y_test[0:fhe_samples]\n",
    "                test_length = fhe_samples\n",
    "\n",
    "            evaluation_result[\"FHE samples\"] = test_length\n",
    "\n",
    "            print(\"Predict (FHE)\")\n",
    "\n",
    "            before_time = time.time()\n",
    "            y_pred_fhe = model.predict(x_test, fhe=\"execute\")\n",
    "            evaluation_result[\"FHE execution time (second per sample)\"] = (\n",
    "                time.time() - before_time\n",
    "            ) / test_length\n",
    "\n",
    "            evaluation_result[\"Accuracy (FHE)\"] = accuracy_score(y_test, y_pred_fhe)\n",
    "\n",
    "    print(\"Done !\\n\")\n",
    "\n",
    "    return evaluation_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the evaluation\n",
    "In the following, we evaluate several types of classifiers : logistic regression, decision tree, random forest and XGBoost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating model LogisticRegression (sklearn)\n",
      "Done !\n",
      "\n",
      "Evaluating model LogisticRegression (Concrete ML)\n",
      "Compile the model\n",
      "Predict (simulated)\n",
      "Predict (FHE)\n",
      "Done !\n",
      "\n",
      "Evaluating model DecisionTreeClassifier (sklearn)\n",
      "Done !\n",
      "\n",
      "Evaluating model DecisionTreeClassifier (Concrete ML)\n",
      "Compile the model\n",
      "Predict (simulated)\n",
      "Predict (FHE)\n",
      "Done !\n",
      "\n",
      "Evaluating model RandomForestClassifier (sklearn)\n",
      "Done !\n",
      "\n",
      "Evaluating model RandomForestClassifier (Concrete ML)\n",
      "Compile the model\n",
      "Predict (simulated)\n",
      "Predict (FHE)\n",
      "Done !\n",
      "\n",
      "Evaluating model XGBClassifier (sklearn)\n",
      "Done !\n",
      "\n",
      "Evaluating model XGBClassifier (Concrete ML)\n",
      "Compile the model\n",
      "Predict (simulated)\n",
      "Predict (FHE)\n",
      "Done !\n",
      "\n"
     ]
    }
   ],
   "source": [
    "results = []\n",
    "\n",
    "# Define the test size proportion\n",
    "test_size = 0.2\n",
    "\n",
    "# For testing FHE execution locally, define the number of inference to run. If None, the complete\n",
    "# test set is used\n",
    "fhe_samples = None\n",
    "\n",
    "# Logistic regression\n",
    "results.append(evaluate(SklearnLogisticRegression(), x_basic, y, test_size=test_size))\n",
    "results.append(evaluate(ConcreteLogisticRegression(), x_basic, y, test_size=test_size))\n",
    "\n",
    "# Define the initialization parameters for tree-based models\n",
    "init_params_dt = {\"max_depth\": 10}\n",
    "init_params_rf = {\"max_depth\": 7, \"n_estimators\": 5}\n",
    "init_params_xgb = {\"max_depth\": 7, \"n_estimators\": 5}\n",
    "init_params_cml = {\"n_bits\": 3}\n",
    "\n",
    "# Determine the type of models to evaluate\n",
    "use_dt = True\n",
    "use_rf = True\n",
    "use_xgb = True\n",
    "predict_in_fhe = True\n",
    "\n",
    "# Decision tree models\n",
    "if use_dt:\n",
    "\n",
    "    # Scikit-Learn model\n",
    "    results.append(\n",
    "        evaluate(\n",
    "            SklearnDecisionTreeClassifier(**init_params_dt),\n",
    "            x_basic,\n",
    "            y,\n",
    "            test_size=test_size,\n",
    "        )\n",
    "    )\n",
    "\n",
    "    # Concrete ML model\n",
    "    results.append(\n",
    "        evaluate(\n",
    "            ConcreteDecisionTreeClassifier(**init_params_dt, **init_params_cml),\n",
    "            x_basic,\n",
    "            y,\n",
    "            test_size=test_size,\n",
    "            predict_in_fhe=predict_in_fhe,\n",
    "            fhe_samples=fhe_samples,\n",
    "        )\n",
    "    )\n",
    "\n",
    "# Random Forest\n",
    "if use_rf:\n",
    "\n",
    "    # Scikit-Learn model\n",
    "    results.append(\n",
    "        evaluate(\n",
    "            SklearnRandomForestClassifier(**init_params_rf),\n",
    "            x_basic,\n",
    "            y,\n",
    "            test_size=test_size,\n",
    "        )\n",
    "    )\n",
    "\n",
    "    # Concrete ML model\n",
    "    results.append(\n",
    "        evaluate(\n",
    "            ConcreteRandomForestClassifier(**init_params_rf, **init_params_cml),\n",
    "            x_basic,\n",
    "            y,\n",
    "            test_size=test_size,\n",
    "            predict_in_fhe=predict_in_fhe,\n",
    "            fhe_samples=fhe_samples,\n",
    "        )\n",
    "    )\n",
    "\n",
    "# XGBoost\n",
    "if use_xgb:\n",
    "\n",
    "    # Scikit-Learn model\n",
    "    results.append(\n",
    "        evaluate(\n",
    "            SklearnXGBoostClassifier(**init_params_xgb),\n",
    "            x_basic,\n",
    "            y,\n",
    "            test_size=test_size,\n",
    "        )\n",
    "    )\n",
    "\n",
    "    # Concrete ML model\n",
    "    results.append(\n",
    "        evaluate(\n",
    "            ConcreteXGBoostClassifier(**init_params_xgb, **init_params_cml),\n",
    "            x_basic,\n",
    "            y,\n",
    "            test_size=test_size,\n",
    "            predict_in_fhe=predict_in_fhe,\n",
    "            fhe_samples=fhe_samples,\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare the models\n",
    "\n",
    "Let's compare the models' performance in a pandas Dataframe. We can see that with only a few bits of quantization, the Concrete models perform as well as their scikit-learn equivalent. More precisely, the small differences that can be observed are only the result of quantization: running the inference in FHE does not impact the accuracy score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>Test samples</th>\n",
       "      <th>n_bits</th>\n",
       "      <th>Accuracy (clear)</th>\n",
       "      <th>F1 (clear)</th>\n",
       "      <th>Precision (clear)</th>\n",
       "      <th>Recall (clear)</th>\n",
       "      <th>max bit-width</th>\n",
       "      <th>Accuracy (simulated)</th>\n",
       "      <th>FHE samples</th>\n",
       "      <th>FHE execution time (second per sample)</th>\n",
       "      <th>Accuracy (FHE)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LogisticRegression (sklearn)</td>\n",
       "      <td>1192</td>\n",
       "      <td></td>\n",
       "      <td>0.824</td>\n",
       "      <td>0.627</td>\n",
       "      <td>0.748</td>\n",
       "      <td>0.606</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>LogisticRegression (Concrete ML)</td>\n",
       "      <td>1192</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.824</td>\n",
       "      <td>0.627</td>\n",
       "      <td>0.748</td>\n",
       "      <td>0.606</td>\n",
       "      <td>18.0</td>\n",
       "      <td>0.824</td>\n",
       "      <td>1192.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DecisionTreeClassifier (sklearn)</td>\n",
       "      <td>1192</td>\n",
       "      <td></td>\n",
       "      <td>0.879</td>\n",
       "      <td>0.783</td>\n",
       "      <td>0.843</td>\n",
       "      <td>0.750</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>DecisionTreeClassifier (Concrete ML)</td>\n",
       "      <td>1192</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.852</td>\n",
       "      <td>0.705</td>\n",
       "      <td>0.818</td>\n",
       "      <td>0.670</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.848</td>\n",
       "      <td>1192.0</td>\n",
       "      <td>0.194</td>\n",
       "      <td>0.848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RandomForestClassifier (sklearn)</td>\n",
       "      <td>1192</td>\n",
       "      <td></td>\n",
       "      <td>0.872</td>\n",
       "      <td>0.761</td>\n",
       "      <td>0.839</td>\n",
       "      <td>0.724</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>RandomForestClassifier (Concrete ML)</td>\n",
       "      <td>1192</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.840</td>\n",
       "      <td>0.645</td>\n",
       "      <td>0.836</td>\n",
       "      <td>0.618</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.84</td>\n",
       "      <td>1192.0</td>\n",
       "      <td>0.295</td>\n",
       "      <td>0.84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>XGBClassifier (sklearn)</td>\n",
       "      <td>1192</td>\n",
       "      <td></td>\n",
       "      <td>0.888</td>\n",
       "      <td>0.806</td>\n",
       "      <td>0.846</td>\n",
       "      <td>0.780</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>XGBClassifier (Concrete ML)</td>\n",
       "      <td>1192</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.841</td>\n",
       "      <td>0.647</td>\n",
       "      <td>0.848</td>\n",
       "      <td>0.619</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.841</td>\n",
       "      <td>1192.0</td>\n",
       "      <td>0.226</td>\n",
       "      <td>0.841</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   name  Test samples n_bits  \\\n",
       "0          LogisticRegression (sklearn)          1192          \n",
       "1      LogisticRegression (Concrete ML)          1192    8.0   \n",
       "2      DecisionTreeClassifier (sklearn)          1192          \n",
       "3  DecisionTreeClassifier (Concrete ML)          1192    3.0   \n",
       "4      RandomForestClassifier (sklearn)          1192          \n",
       "5  RandomForestClassifier (Concrete ML)          1192    3.0   \n",
       "6               XGBClassifier (sklearn)          1192          \n",
       "7           XGBClassifier (Concrete ML)          1192    3.0   \n",
       "\n",
       "   Accuracy (clear)  F1 (clear)  Precision (clear)  Recall (clear)  \\\n",
       "0             0.824       0.627              0.748           0.606   \n",
       "1             0.824       0.627              0.748           0.606   \n",
       "2             0.879       0.783              0.843           0.750   \n",
       "3             0.852       0.705              0.818           0.670   \n",
       "4             0.872       0.761              0.839           0.724   \n",
       "5             0.840       0.645              0.836           0.618   \n",
       "6             0.888       0.806              0.846           0.780   \n",
       "7             0.841       0.647              0.848           0.619   \n",
       "\n",
       "  max bit-width Accuracy (simulated) FHE samples  \\\n",
       "0                                                  \n",
       "1          18.0                0.824      1192.0   \n",
       "2                                                  \n",
       "3           4.0                0.848      1192.0   \n",
       "4                                                  \n",
       "5           4.0                 0.84      1192.0   \n",
       "6                                                  \n",
       "7           4.0                0.841      1192.0   \n",
       "\n",
       "  FHE execution time (second per sample) Accuracy (FHE)  \n",
       "0                                                        \n",
       "1                                  0.001          0.824  \n",
       "2                                                        \n",
       "3                                  0.194          0.848  \n",
       "4                                                        \n",
       "5                                  0.295           0.84  \n",
       "6                                                        \n",
       "7                                  0.226          0.841  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option(\"display.precision\", 3)\n",
    "\n",
    "results_dataframe = pd.DataFrame(results)\n",
    "results_dataframe.fillna(\"\")"
   ]
  }
 ],
 "metadata": {
  "execution": {
   "timeout": 10800
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
